<html>
<head><meta charset="utf-8"><title>Reusing cached rmetas for pipelined builds? · t-cargo · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/index.html">t-cargo</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Reusing.20cached.20rmetas.20for.20pipelined.20builds.3F.html">Reusing cached rmetas for pipelined builds?</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="243323275"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Reusing%20cached%20rmetas%20for%20pipelined%20builds%3F/near/243323275" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jeremy Fitzhardinge <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Reusing.20cached.20rmetas.20for.20pipelined.20builds.3F.html#243323275">(Jun 20 2021 at 21:46)</a>:</h4>
<p>Hey all - I've been looking at pipelined builds a bit more deeply, and I'm wondering if there's some potential performance left on the table.</p>
<p>As I understand it, if you do a pipelined <code>cargo build</code> it will invoke <code>rustc --emit link,metadata</code>, and use the artifact notification mechanism to kick off dependent builds as soon as the metadata is available. </p>
<p>In principle this is fine in terms of CPU cost, since the front-end work will always have to be done, so the only cost of <code>--emit ...,metadata</code> is that of serializing and writing it. But there's still a latency cost as the dependent builds are still blocked until the rmeta is written.</p>
<p>But if we already have rmeta files from previous builds cached - or even from previous <code>cargo check</code> runs - then would that allow the dependent builds to be kicked off earlier, getting more potential parallelism?</p>
<p>A to be transparent, I'm interested in making <code>rustc --emit metadata</code> generate the same "full fat" rmeta files that it does when used in combination with <code>--emit metadata,link</code>. I've measured this as having ~8% perf cost when running <code>cargo check</code> on Cargo itself (<a href="https://internals.rust-lang.org/t/rfc-emit-check/14711/5?u=jsgf">https://internals.rust-lang.org/t/rfc-emit-check/14711/5?u=jsgf</a>), and I'm wondering if there's ways in which this cost can be amortized.</p>
<p>The best case for this proposal is if you've already done a <code>cargo check</code> build and a subsequent <code>cargo build</code> is a full rebuild. For partial rebuilds there's going to be less effect, but on the other hand the perceived cost of an incremental cargo check would also be smaller. I'm not sure how it interacts with the perf effects of incremental builds.</p>



<a name="243323615"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/246057-t-cargo/topic/Reusing%20cached%20rmetas%20for%20pipelined%20builds%3F/near/243323615" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/246057-t-cargo/topic/Reusing.20cached.20rmetas.20for.20pipelined.20builds.3F.html#243323615">(Jun 20 2021 at 21:54)</a>:</h4>
<p>(deleted)</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>